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[57] ABSTRACT 

An image data registration system and method of storing a 
first data set of three-dimensional image data associated with 
a predetermined portion of an object with reference to a first 
coordinate firame; obtaining and storing a second data set of 
three-dimensional image data associated with a surface of 
the predetermined portion of the object with reference to a 
second coordinate firame; obtaining and storing a third data 
set of three-dimensional image data associated with a probe 
in the vicinity of the predetermined portion of the object 
with reference to a third coordinate firame; and registering 
the first, second and third data sets to generate a matched 
image data set in which the first, second and third coordinate 
frames are relatively aligned with one another. 
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SYSTEM AND METHOD OF REGISTRATION 
OF THREE-DIMENSIONAL DATA SETS 

This is a continuation-in-part of application Ser. No. 
08/299378 filed on Sep. 1, 1994 now U.S. Pat. No. 5,531, 5 
520. 

This invention was made with government support under 
Grant Number N00014-91-J-4038 awarded by the Navy. 
The government has certain rights in the invention. 

The invention relates in general to a system and method 10 
of image data registration, and more particularly to a system 
and method of registering three-dimensional surgical image 
data utilized in image guided surgery and frameless stereo- 
taxy. 

Neurosurgical procedures, such as biopsy or tumor 15 
extraction, require highly precise localization on the part of 
the surgeon, in order to attain the desired extraction of 
material while minimizing collateral damage to adjacent 
structures. The problem is exacerbated by the fact that the 
localization is three dimensional in nature, and often 20 
requires localizing a structure deeply buried within the 
cranium. While methods exist (e.g. MRI, CT) for imaging 
and displaying the 3D structure of the cortex, this still leaves 
the surgeon with the problem of relating what she sees on the 
3D display with the actual anatomy of the patient. 25 

Conventional solutions typically involve presurgically 
attaching a stereotactic frame to the patient's skull, then 
imaging the skull and frame as a unit. This allows the 
surgeon to locate, from the 3D images, the location of the 
tumor or other target relative to a coordinate system attached 30 
to the stereotactic frame, and thus to the patient's head. As 
well, the frame typically includes a movable armature that 
allows the positioning of a probe at any orientation relative 
to the patient. This lets the surgeon mark a planned angle of 
entry to access the tumor, thus localizing the expected 35 
extraction of material. 

Unfortunately, the use of stereotactic frames is both 
cumbersome to the surgeon, and involves considerable dis- 
comfort to the patient, who must wear the device for several 
days between imaging and surgery. In addition, such frames 40 
can have limited flexibility, especially should surgical plans 
have to change in the middle of the procedure, e.g. if the line 
of attack is found to pass through critical regions, such as the 
motor strip. 

In addition to the aforementioned drawbacks of conven- 45 
tional surgical techniques, it is often necessary for surgeons 
to insert rigid probes or instruments into bodily cavities for 
certain procedures. Typically, such insertions are required 
for removal of material from the interior structure of the 
patient, without the need for extensive surgical invasion of 50 
the patient. Unfortunately, it is difficult to view the probe 
with respect to the internal parts of the cavity in which the 
instrument is inserted. 

Accordingly, it is an object of the present invention to 
provide an imaging method and system which registers sets 55 
of three-dimensional image data of an object. 

It is another object of the present invention to provide a 
surgical imaging method and system which registers clinical 
data, such as segmented MRI or CT reconstructions, with 
surface data associated with a portion of a patient's body. 60 

It is yet another object of the present invention to provide 
a surgical imaging method and system which generates 
real-time, adaptive, enhanced visualizations of the patient in 
the operating room so as to accommodate dynamic image- 
guided surgical planning and image guided surgical 65 
procedures, such as biopsies or minimally invasive thera- 
peutic procedures. 



2 

It is still another object of the present invention to 
provide a surgical imaging method and system which estab- 
lishes visual feedback to the surgeon to accommodate view- 
ing the position of the instrument in the patient, thus 
enhancing the ability to guide further insertions and proce- 
dures with the rigid instrument. 

SUMMARY OF THE INVENTION 

The present invention provides an image data registration 
system and method of storing a first data set of three- 
dimensional image data associated with a predetermined 
portion of an object as model image data with reference to 
a first coordinate frame; obtaining and storing a second data 
set of three-dimensional surface image data associated with 
a surface of the predetermined portion of the object with 
reference to a second coordinate frame; obtaining and stor- 
ing a third data set of three-dimensional probe image data 
associated with a probe in the vicinity of the predetermined 
portion of the object with reference to a third coordinate 
frame; and registering the first, second and third data sets to 
generate a matched image data set in which the first, second 
and third coordinate frames are relatively aligned with one 
another. 

The present invention more particularly provides a sur- 
gical method and system which performs the registration of 
clinical sensory data with the corresponding position of the 
patient's body on the operating table at the time of surgery, 
and with surgical instruments, using methods from visual 
object recognition, which do not require the use of a 
previously attached stereotactic frame. The method has been 
combined with an enhanced visualization technique, in 
which there is displayed a composite image of the 3D 
anatomical structures and probes with a view of the patient's 
body. 

This registration enables the transfer to the operating 
room of preoperative surgical plans, obtained through analy- 
sis of the segmented 3D preoperative data, where they can 
be graphically overlaid onto video images of the patient. 
Such transfer allows the surgeon to apply carefully consid- 
ered surgical plans to the current situation and to mark 
landmarks used to guide the progression of the surgery. 
Extensions of the method and system include adaptively 
re-registering the video image of the patient to the 3D 
anatomical data, as the patient moves, or as the video source 
moves. 

The file of this patent contains at least one drawing 
executed in color. Copies of this patent with color drawing 
(s) will be provided by the Patent and Trademark Office 
upon request and payment of the necessary fee. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 shows a block diagram of an image data registra- 
tion system in accordance with the present invention; 

FIG. 2 shows a flow outline of the data registration 
technique utilized in accordance with the present invention; 

FIG. 3 shows exemplary results of registering laser data 
derived from laser scan lines of a head to the skin surface 
segmented from an MRI data set; 

FIG. 4 shows exemplary final results of the system of the 
present invention as a combined video/MRI visualization in 
which a surgeon is provided with an enhanced visualization 
view inside of the head of a patient; 

FIG. 5 shows a flowchart of the data registration process 
in accordance with the present invention incorporating a 
trackable surgical probe in accordance with the present 
invention; 
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FIG. 6 shows a block diagram of an exemplary image data 
registration system incorporating a trackable probe; and 

FIG. 7 shows exemplary final results of the system of the 
present invention as a combined visualization in which a 
surgeon is provided with an enhanced view inside of the 
head of a patient and the position of the tip of the trackable 
probe. 

DETAILED DESCRIPTION OF THE 
ILLUSTRATED EMBODIMENTS 

For purposes of illustration, an exemplary embodiment of 
the present invention will be described with reference to a 
craniotonomy procedure. It will be appreciated that the 
described image registration method and system of the 
present invention is not limited to use only with cranial 
structures, but in fact may be used for registering three- 
dimensional image data for sets of other objects including 
other portions of an anatomical body. 

Accordingly, with reference to FIG. 1, there is shown an 
image data registration system 100 in accordance with the 
present invention. The system 100 operates in accordance 
with the following exemplary overview for cranial surgery. 
A patient requiring surgical therapy is initially scanned by a 
three-dimensional, high resolution, internal anatomy 
scanner, such as Magnetic Resonance Imaging (MRI) or 
Computed Tomography (CT). It will be appreciated that any 
form of volumetric imaging techniques, such as PET, 
SPECT, etc., can also be used. If a brain atlas is available, 
or if a previous, segmented scan of the patient is available, 
the current scan is registered to this prior scan, and the match 
is used to drive fast, accurate, automated segmentation of the 
current scan, and to identify changes since the reference scan 
was taken, to better identify the pathology. If no previous 
information is available, then the current scan is segmented 
to produce organ surface data and other anatomical struc- 
tures such as ventricles and tumor tissue using conventional 
automated algorithms. This is typically done by training an 
intensity classifier on a user selected set of tissue samples, 
where the operator uses knowledge of anatomy to identify 
the tissue type. Once initial training is completed, the rest of 
the scans can be automatically classified on the basis of 
intensities in the scanned images, and thus segmented into 
tissue types. Conventional automatic methods for removing 
gain artifacts from the sensor data can be used to improve 
the segmentation. This 3D anatomical reconstruction is 
referred to as the model, and is represented relative to a 
model coordinate frame. For simplicity, the center of the 
coordinate system can be taken as the centroid of the points. 

The patient is then placed in an operating room, which is 
equipped with a laser range scanner for obtaining depth data 
of the patient's skin surface where the surgery is to be 
performed; and enhanced visualization equipment, such as a 
video or digital camera, mixer and display monitor, a 
head-mounted display with trackable landmarks, an operat- 
ing room microscope with video projection overlay feed, 
along with microscope-mounted trackable landmarks, or 
transparent projection screens along with screen mounted 
trackable landmarks, medical instrument holders containing 
trackable landmarks. The operating table may also contain 
fixed raised landmarks that will remain viewable and in the 
same position during surgery, and landmark tracking equip- 
ment. 

Prior to draping, the patient is scanned by the laser range 
scanner The 3D locations of any table landmarks are also 
calculated to identify their location relative to the patient. 
The current MRI or CT scan is automatically registered to 
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the patient skin surface depth data obtained by the laser 
range scanner. This provides a transformation from MRI/CT 
to patient. The position and orientation of a video camera 
relative to the patient is determined by matching video 

5 images of the laser points on an object to the actual 3D laser 
data. This provides a transformation from patient to video 
camera. ITie registered anatomy data is displayed in 
enhanced visualization to "see" inside the patient. In 
particular, the two previously computed transformations can 
0 be used to transform the 3D model into the same view as the 
video image of the patient, so that video mixing allows the 
surgeon to see both images simultaneously. Alternatively, 
the images are combined with a surgical microscope or 
transparent imaging panel in order to augment the line-of- 

15 sight view of the surgeon with the MRI data. The patient is 
draped and surgery is performed. 

The enhanced visualization does not interfere with the 
surgeon, nor does it require any procedures different from 
that to which the surgeon is accustomed. Rather, the system 

20 provides the surgeon with additional visualization informa- 
tion to greatly expand the limited field of view. The location 
of table landmarks can be continually tracked to identify 
changes in the position and attitude of the patient's head, 
relative to the visualization camera. Visualization updates 

25 are performed by re-rendering based on this tracking. 
Viewer location is continually tracked to identify any 
changes in the position of the viewer. In the case of a 
stationary video camera, this is unnecessary, though in the 
case of head-mounted displays such tracking is necessary. 

30 Visualization updates are performed by re-registration. If 
landmarks are used for tracking, re-registration is unneces- 
sary. Updates are performed simply by re-rendering based 
on the tracked position information. Medical instruments 
may be tracked to align them with predetermined locations 

3S as displayed in the enhanced visualization. In general, the 
surgical procedure is executed with an accurately registered 
enhanced visualization of the entire relevant anatomy of the 
patient, and thus with reduced side effects. 

With reference back to FIG. 1, the patient's body 102 is 

40 positioned on the operating table 104. The system 100 
obtains three-dimensional image data from the skin surface 
of the patient with the utilization of a 3D laser scanning unit 
106, which includes both a laser striping device 108 and a 
laser camera 110. Exemplary apparatus for carrying out the 

45 laser striping is described in U.S. Pat. Nos. 4,498,778 and 
4,628,469, incorporated herein by reference. In essence, the 
system utilizes uses a plane of laser light and a video camera 
to obtain three dimensional measurements of the patient's 
skin, and uses the "structured light" method of obtaining the 

50 desired measurements. This method is based on the principal 
of triangulation for measurement. 

The 3D laser scanning unit 106 controls the laser striping 
device 108 to generate a laser beam which is optically spread 
out to form a plane of light 112, which is projected in front 

55 of the laser camera 110 at an angle to the optical axis of the 
camera. The plane of light is formed, for example, by a laser 
beam reflected off of an oscillating mirror or a passive lens 
as is well known. 
The laser camera 110 is placed at an angle to the plane of 

60 light such that a portion of the plane is in the camera field 
of view (FOV). As this plane of light strikes an object, such 
as the patient's skin, the diffuse reflection appears on the 
video camera image as a line of light 114. In other words, 
when an object is placed in this visible region such that it 

65 intersects the laser plane, points in the camera image plane 
illuminated by the laser unambiguously correspond to fixed 
3D scene points. This is most easily appreciated by consid- 
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ering the case of objects on a flat support plane, with the 
plane of laser light striking the support plane at an oblique 
angle. When only the support plane is imaged, the laser 
plane makes a straight line in the image, and the camera can 
be oriented so that this line is vertical, for example. When an 
object is placed on the support plane, the imaged intersection 
of laser plane and the object is deflected from the previously 
recorded position, by an amount that is a direct function of 
the height of the object above the plane. By measuring this 
deflection, the distance to the observed point can be com- 
puted. 

The projection of the stripe in the plane of the illumination 
onto the focal plane of the imaging device is unique. Each 
sample point in the image plane corresponds to a unique 
point in the plane of the laser illumination. This correspon- 
dence can be determined through a transform which, in turn, 
can be determined by scanning a known 3D shape. The laser 
scanning unit uses this unique mapping between image 
plane points and 3D points in space to determine the 3D 
coordinates points on the surface of the patient's skin 
illuminated by the laser. When the laser is moved, a different 
cross-section of the skin under the scanner can be measured. 
With multiple scans and the acquisition of multiple points 
within each scan, a sample grid is converted into 3D 
measurements. The density is only limited by the number of 
scans taken and the number of samples in each scan taken 
from the scanning unit. In the surgery example, approxi- 
mately 20 scans are taken with between 100 and 200 3D 
points measured in each. 

The laser scanning unit 106 used could be substituted for 
by any of a number of surface 3D scanners. There are 
numerous other conventional methods, including laser radar 
and moire fringe analysis, that could be used in the system 
100. Other non-contact sensor types, including ultrasound or 
radar, are possible, as are a wide array of contact (probing) 
types of measurement devices. All that is required is the 
derivation of a modest number of accurate skin surface 3D 
points. 

The system 100 also includes a real-time video camera 
116 for providing a real-time image of the patient to which 
the 3D anatomy data will be registered. In one embodiment, 
the real-time video camera may be the same camera as the 
laser camera 110. An image data processor 118 serves to 
register the 3D surface data to the 3D anatomy data which 
has been prestored in an image data storage unit 120. A scan 
converter 122 is used to take the terminal outputs of the 
processor and convert them into video signals. Some com- 
puter processors provide direct video outputs, and thereby 
wouldn't require a scan converter. In an exemplary 
embodiment, the image data processor is an IBM RS6000 or 
IBM PVS used in conjunction with a Sun Sparc 10. A video 
mixer 124 mixes the video images from the processor, 
scanning unit and real-time video camera and thereafter fed 
to a video monitor 126 for live video visualizations 
enhanced with conventional computer graphics. 

Prior to using the aforementioned hardware in a surgical 
setting, a calibration of the system is desirable. Calibration 
of the laser scanning unit 106 including the laser striping 
unit 108 and the cameras 110, 116 is performed using scan 
data from a precisely machined shape referred to as a gauge. 
This known shape, along with the images of the laser scans 
from it, can be used to precisely calibrate the laser instru- 
ment such that all subsequent measurements, anywhere in 
the operating range of the scanning unit will result in 
accurate 3D measurements as measured relative to some 
fixed reference frame. Since this reference frame is arbitrary, 
vertices of the laser system calibration gauge are used to 
define the frame. 
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The real-time video camera is also calibrated using the 
same gauge. The camera calibration is used to determine the 
appropriate viewpoint for rendering the registered 3D 
images prior to being mixed with the video images. The 

5 calibration determines the focal length of the camera lens, 
the position of the focal point, and the orientation of the 
image plane relative to the 3D laser reference frame. Since 
the laser calibration provides a mapping between every 2D 
point on the image plane and their corresponding 3D point 

10 on the laser plane, each measurement from objects in front 
of the scanning unit provides such a image/laser plane point 
correspondence and can be used to find the camera model. 

In the case where the laser camera is used as the real-time 
video camera, since it is fixed relative to the scanner, an 

15 initial estimate of the camera position is based on the 
geometry of the setup. The calibration involves the follow- 
ing steps: 1) project the measured 3 D points onto the image 
plane using the latest estimate of the camera model; 2) 
determine the summation of squared distances between each 

20 2D projection from the previous step and the known 2D 
image points using the laser measurements; and 3) modify 
the camera model, preferably using the conventionally 
known Powell's method, described by Press et al., Numeri- 
cal Recipes in C, The Art of Scientific Computings Second 

25 Edition, Cambridge University Press, 1992, incorporated 
herein by reference, to minimize the error measure from step 
2. Once the camera model has been adjusted to minimize the 
reprojection error until the aggregate error is less than some 
predefined threshold, the camera calibration is complete. 

30 The data registration technique utilized in accordance 
with the present invention is described hereinafter with 
reference to the flow outline of FIG. 2. The rigid 3D — 3D 
registration technique in its general form consists of the 
following input and output. The input comprises two 3D 

35 surface data sets, represented as sets of 3D points, each in its 
own coordinate frame. In the illustrated exemplary 
embodiment, one of these data sets would be from the laser 
scanner, and the other would be from a segmented skin 
surface of a medical scan, such as CT or MRI. other 

40 embodiments , not relying on the laser scanner, are possible, 
however, and are described below. The points are assumed 
to lie on the same structural surface, although the coverages 
of the points do not need to exactly overlap and outliers may 
be present. The output involves a six degree-of-freedom 

45 rigid body transformation mapping one of the data sets into 
the other, or equivalently, transforming the coordinate frame 
of one data set into the other. Such six degree-of-freedom 
methods solve for three translation and three rotation param- 
eters for matching the two data sets. The method is based on 

50 a hierarchical solution approach in which coarse initial 
alignments are generated using coarse resolution data and 
then refined using a series of optimization steps to guide the 
solution towards the best match. 

Although the system 100 will operate without such 

55 information, it is convenient to specify a body axis and a 
nose axis, both in model coordinates, and in laser coordi- 
nates. Typically an axis from the center of the head through 
the top of the skull is taken as the body axis, and an axis from 
the center of the head through the nose is taken as the nose 

60 axis. All of the axes need only be roughly accurate. The 
model axes should be given in terms of model coordinates, 
and the data axes should be provide in terms of laser 
coordinates. Accordingly, the operator can perform an initial 
coarse alignment of the data. This initial alignment does not 

65 need to be highly accurate. Rotational errors on the order of 
10-20 degrees and translational errors on the order of 
centimeters are permissible. 
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Id the initial data matching method of the registration method is used to align the data sets by first translating the 

process (step 201), the image data processor 118 operates to second data set so that its centroid aligns with the centroid 

generate a coarse alignment of the two 3D data sets to use of the first data set. Then the second data set is rotated so that 

as a starting point for subsequent refinement. The data its principal directions (moments of inertia) align with the 

matching method utilizes several stage approaches for 5 principal directions of the first data set. These directions are 

accomplishing the initial match, including contextual esti- computed by taking the eigenvectors of the inertia matrix of 

mation (step 202), centroid/axes alignment (step 203), and eacn ( j ata ^ 
sampled constrained search (step 204). 

TTie contextual estimation stage relies on an operator's Id the sam P led constrained search stage, if there is limited 

knowledge of the rough pose of patient with respect to the 10 overlap in the coverage of the two data sets (i.e., if there are 

scanner and the operating room for use to estimate an 001 both relatively complete models of the anatomy), a 

alignment. If rough body and nose axes estimates are sample is taken of a small number (e.g. three) of widely 

known, the operator can use these together with knowledge spaced points from one of the data sets, and then using an 

of the view direction of the laser camera 110 to estimate a interpretation tree search as described in Grimson, Object 

rough view direction of the model or object being observed. 15 Recognition by Computer: The Role of Geometric 

One such method involves the user specifying a rough view Constraints, MIT Press, 1990, incorporated herein by 

( e g- right side up, left side up, etc.) and then specifying a reference, those sampled points are matched to data points 

rough orientation, in that view, of the nose and/or body axis. in the other data set. The interpretation tree search method 

This can be used to compute an initial alignment of the body basically searches over all possible ways of matching small 

relative to the MRI or CT data. 20 ^ of ^ f eatures fr om me ^ 0 <j ata ^ p or eacn pairing 

In an exemplary case, given the rough view, a sampled set of data features from the two data sets, the method tests 

of visible points of the laser line 114 is extracted using a whether the pairwise distances between points are roughly 

z-buffer. In particular, given a pixel size for the z-buffer and the same. If all such tests are valid, the match is kept, and 

given an orientation for that buffer, a projection of all of the the coordinate frame transformation that maps the data 

model points into this array is made. Within each pixel, only 25 points from the first set into their corresponding points in the 

the point closest to the viewer is kept. This action provides second set is computed, 

a temporary model, which can be used for matching. It will m , ^ , „ . 

be appreciated that even if body and nose axes estimates are „ ^ d f 5cribed /^formations f°™ * of hypotheses. 

available, they are usually not sufficiently accurate to define Due t0 the ^P 1 ^ ° f ^ act " al corresponding 

the final solution 30 P otDls ma y not exist > me hypothesized transformations 

„ * 1 , are at best approximations to the actual transformation. An 

It estimates or the axes are not available to the operator. . , < - , . IT „ . , # t 

, c A c . f iL j * ■ . f . alignment method, as described in Huttenlocher et al., 

then a sample of a set of views of the model is taken, by objects ^ ,u lHrvnctU mh an Irnage< 

sampling a set of evenly spaced directions on the view Inter ^ ation | 1 j ollrnal Computer Vision, 5 (2), 1992, pp. 

sphere^ For each view, the z-buffer method described above 35 195 _ 212 ^0^^ hcrein by re f erence , is used to filter 

is used to extract a sampled set of visible points of the 4 , , a. a -j- t c i_ l. *u * 

moc j e j Y v these hypotheses. Accordingly, for each hypothesis, a veri- 
fication is made that the fraction of the laser points, trans- 

Thereafter, a graphical interface with the image data formed by the hypothesized transformation, without a cor- 

processor is used which enables the operator to guide the responding model point within some predefined distance is 

two data sets into rough initial alignment. In the exemplary ^ less than so me predefined bound. Those hypotheses that fail 

embodiment, the interface provides the operator with two mis verification are discarded. For efficiency, two levels of 

stages for alignment. The first stage presents an image of the sampling of the laser points are used, first verifying that a 

patient as seen from the laser camera 110 on top of which is coa rsely sampled set of laser points are in agreement, then 

superimposed the positions of the laser data points, as seen farthcr verifying, for those that pass this test, that all the 

in that view. The operator employs a conventional computer 45 laser points are in agreeme nt. 
mouse, for example, to delineate those laser points that are 

to be included as part of the data set. This is done, for Th e image data processor next operates to perform an 

example, by drawing bounding boxes around the sets of interpolated refinement method (step 205) which includes 

points to be included. me stages of Gaussian-weighted transformation evaluation 

Once the data has been selected, a second graphical 50 < step 20 ?' PoweU,s met ! lod of ° pdm j^ ion ( ste P 207 )> and 

interface presents three orthogonal views of the anatomical ^creased Gaussian resolution (step 208). 

3D data set, for example the 3D MRI data, together with the For each verified hypothesis, the image data processor 

selected laser data points on the video monitor 126. The performs an initial refinement aimed at guiding the regis- 

operator can again edit the laser points, in this case using tration in the general direction of the global error minimum, 

bounding boxes to indicate points to be excluded from the 55 To perform this refinement an evaluation is made of the 

data set. Once the data has been filtered in this way, the current pose by summing, for all transformed points (from 

computer mouse is employed to manipulate the laser data data set 2), a term that is itself a sum of the distances from 

relative to the anatomical 3D data. In particular, the interface the transformed point to all nearby reference surface points 

is used to translate (in 3D) the laser data relative to the MRI (data set 1), where the distance is weighted by a Gaussian 

data, and to rotate the laser data about any of the three axes 60 distribution. This Gaussian weighted distribution is a 

of the orthogonal views. The result of these operations is that method for roughly interpolating between the sampled ref- 

the laser data may be translated and rotated arbitrarily in ere nee points to estimate the nearest point on the underlying 

order to bring the two data sets into rough alignment. The surface to the transformed data point. More precisely, if 1, is 

output of the stage, upon completion, is a rough six -degree- a vector representing a data point, m / is a vector representing 

of-freedom transformation of one data set into the other. 65 a reference point, and T is a coordinate frame 

With respect to the centroid/axes alignment stage, if the transformation, then the evaluation function for a particular 

two data sets (almost) completely overlap each other, a pose (or transformation) is shown in the following equation: 
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from MRI or CT to derive the second data set, the method 

\Tii-mj\ 2 CD is not restricted to such data inputs. For example, in change 

£l {T) ~ ~2j 2j e detection, the goal is to measure differences between MRI or 

' J CT scans taken of the same patient at different times. In this 

5 case, the registration technique takes as input two different 

This objective function can be visualized as if a Gaussian MRI dala sets > and registers sampled points from the same 

distribution of some spread a is placed at each reference anatomical surface (e.g. the intercranial cavity). The results 

surface point, then summed with the contributions from each cao then be ^ to ali S n an MRI wth the first such 

such distribution at each point in the volume. The contri- scan > t0 "^section the aligned scan so that the slices corre- 

bution of each transformed data point towards the evaluation 10 s P° nd t0 the sIice P lanes of the oli ® nal ^ ^ then to take 

function is simply the summed value at that point. Because una g e differences between the original slices and the resec- 

of its formulation, the objective function is generally quite tioned ab *e ned slices * Such ima 8 e differences can be used to 

smooth, and thus facilitates pulling in solutions from mod- evaluate changes in structures such as lesions or tumors, for 

erately removed locations in parameter space. This evalua- example, to measure the effectiveness of drug or radiation 

tion function is iteratively minimized using the convention- 15 b ? measuring the change in the size of the structure 

ally known Powell's method, described in the previously after ^ c&tmcnt 

cited publication of Press et al. The result is an estimate for Given ^ 0Ut P ut of thc registration stage, and the seg- 

the pose of the second data set in the coordinate frame of the mented MRI or CT data, the transformation stage (step 213) 

first data set simply applies the rigid coordinate frame transformation to 

rjy% j , o . , , 4 . . 20 all of the data, bringing it into the coordinate frame of the 

The described refinement and evaluation process is , , . . . . « , „ tl _ 

4 j . u - i , cr^ ' i ■** n laser data. This transformed data set is then passed to the 

executed using a multiresolution set of Gaussians. Initially, ., . . „ _ - ... . \. - 

ujuj>. •■ j*n*a i video mixing stage. For purposes of illustration, FIG. 3 

a broad based Gaussian is used to allow influence over large . i i* * • „ • 1 j * j • ^ 

w . . , , ... f shows exemplary results of registering laser data derived 

areas, resulting m a coarser alignment, but one which can be - , . . * i_ j *l t • ^ j 

• i f .j r * *• from laser scan Imes of a bead to the skin surface segmented 

reached from a wide range of starting positions. - wn TJ . , ™ i t- / j \ 

c. . tl ,7 , ^ . i • . 25 from an MRI data set. The laser scan lines (red curves) are 

Subsequently, more narrowly tuned Gaussian distributions , , XMTtr , - / . . v ,. 4 v f / iL 

u j . a *u u-i c • t t_ shown overlaid on the MRI skin (shown in white) after the 

can be used to refine the pose, while focusing on only nearby . . . , , , Vt_ . . ' 

j » - ♦ * j . * u ° J J registration process is completed. The anatomical structures 

data points to derive the pose. 4 , f , MTiJ , . , , ... 

^ r . . A . 0 segmented from the MRI scan are depicted with a tumor 

The image data processor next operates to perform a bei and ventride ^ Wue 

detailed refinement method (step 209) which includes the J0 ^ of the ffi takgs ^ , t a transformed 

stages of least squares transformation evaluation (step 210), model of ^ j^., anatomyj ( MRI of ^ where the 

Howell s method ot optimization (step 211), and random transformation has br0UBn , toe model 

into alignment with 

transformation perturbation (step 212). me actual of me p ^ Qi {n ^ ^ coordinate 
Based on the resulting pose of the interpolated refinement, &ame> ^ a ^ ^ view of the patient taken from the laser 
the pose evaluation process is repeated using a rectified least 3S camera u0 associated with th e laser scanning system, 
squares distance measure. Each pose is evaluated by mea- Because the transformed model is now in the coordinate 
suring the distance from each transformed data point to the framc of ^ laser system> which is measurcd rclative t0 the 
nearest reference surface point (with a cutoff at a predefined cam era, the image data processor can straightforwardly 
maximum distance to guard against outliers or missing project the model into the plane of the video image, creating 
data). The pose evaluation is the sum of the squared dis- ^ a image of ±c model image can then be mixed 
lances of each data point. Powell's method is again used to ^ the Uve video image of the patient taken by the 
find the least-squares pose solution. Here the evaluation rea l-time video camera 116, to provide an enhanced visual- 
function is shown in the following equation: ization. 

For illustrative purposes, FIG. 4 shows exemplary final 

Ez(T) = ^ min/rf 2 ^ \Tti - m l 1 } ® 45 resu ^ ts °^ me system 100 as a combined video/MRI visual- 

L-A ^ \ j J f ization in which a surgeon is provided with an enhanced 

visualization view inside of the head of a patient. The 
registration transformation computed between the laser and 

where d max is a preset maximum distance. The expectation MRI coordinate frames as shown in FIG. 3, directly supports 

is that this second objective function is more accurate 50 the visualization from the viewpoint of a calibrated camera, 

locally, since it is composed of saturated quadratic forms, The tumor is shown in green and ventricle in blue, 

but it is also prone to getting stuck in local minima. The video mixing can be done for any selected portion of 

In order to avoid such local minima, the solution is the anatomy of the model. In one embodiment, only the 

randomly perturbed and subjected to a repeat of the least tumor is overlaid on the video image. In this case, the 

squares refinement. The observation is that while the above 55 surgeon can use the alignment, as viewed on a video screen, 

method always gets very close to the best solution, it can get to mark surgical plans on the patient's scalp, e.g. recording 

trapped into local minima in the minimization of E2. the position of the tumor from that viewpoint, marking 

Accordingly, this perturbation and refinement process is locations for making the incision to fold back a flap of skin, 

continued, keeping the new pose if its associated E2 value marking positions for making the craniotomy, etc. In alter- 

is better than the current best solution. The process is 60 native embodiments, the video output can be provided to a 

terminated when the number of such trials that have passed head-mounted display device, a surgical microscope having 

since the E2 value was last improved becomes larger than a a video overlay feed, or a transparent projection screen, 

predetermined threshold. The final result is a pose, and a The registration process described heretofore is also 

measure of the residual deviation of the fit to the reference extended to facilitate the tracking of surgical instruments 

surface. 65 and probes. An exemplary scenario involves the insertion of 

While the exemplary embodiment involves using laser a rigid instrument into the sinus cavity of a patient. The 

scanning to derive one data set, and segmented skin surfaces primary goal of such an insertion, typically, is to achieve 
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removal of material from an interior structure of the patient, 
without the need for extensive surgical invasion of the 
patient. 

Due to the fact that the surgeon's view is often obstructed 
by parts of the patient and by the surgeon's hands, it is 5 
desirable to provide a type of visual feedback to the surgeon 
to accommodate viewing the position of the instrument in 
the patient, thus enhancing the ability to guide further 
insertions and procedures with the rigid instrument. The 
process involves registering the three-dimensional anatomi- 10 
cal data (MRI, CT, etc.) to the three-dimensional skin 
surface data, which corresponds to the position of the 
patient; registering data related to a tradable probe to the 
three-dimensional skin surface data; and thereafter utilizing 
the data registration between the probe and the skin surface 15 
to provide visualization information to the surgeon. 

With reference to FIGS. 5 and 6, an exemplary alternative 
embodiment of the present invention including data regis- 
tration and visualization with respect to a trackable probe is 
presented. FIG. 5 shows a flowchart of the data registration 20 
process in accordance with the present invention incorpo- 
rating a trackable surgical probe in accordance with the 
present invention. FIG. 6 shows a block diagram of an 
exemplary image data registration system incorporating a 
trackable probe. 25 

Initially, (step 501) a medical scan of the patient, e.g. 
MRI, CT, is acquired and segmented into anatomical struc- 
tures as previously described. The three-dimensional 
anatomy image data is stored in storage 120. Thereafter, 
(step 502) the stored anatomy image data is then registered 30 
to the skin surface image data acquired by the laser scanning 
unit 106 as described in detail with reference to FIGS. 1 and 
2. 

At this point in the process, the probe 600 is tracked so as 
to determine both its position and orientation with respect to 35 
the patient (step 503). The probe to be tracked is equipped 
with a set of markers 602 which are preferably positioned at 
an end of the probe that will remain outside the body cavity. 
The markers are, for example, visual patterns or infrared 
light emitting diodes (IR-LEDs). The markers are tracked by 40 
a set of tracking cameras 604, 606. The cameras forward the 
measured positional information of the markers to a probe 
tracking unit 608, which in turn operates to determine the 
position and orientation of the probe relative to a coordinate 
frame associated with the cameras and tracking unit. It will 45 
be appreciated by those of skill in the art that a generic 
system for tracking visual markers is commercially available 
as the Pixsys Tracking System manufactured by Image 
Guided Therapy Inc. or the Optotrack Tracking System 
manufactured by Optotrack Inc. 50 

The coordinate frame utilized by the tracking unit is then 
calibrated to the coordinate frame of the laser scanning unit 
106 (step 504) by the image data processor 118 which 
initially measures and matches a smalt set of points from 
each data set as previously described. Topically this cali- 55 
b rati on is carried out by measuring a set of points on a 
calibration object with both the laser and trackable probe. 

The surgeon will typically proceed by inserting the probe 
600 or instrument into the body cavity. The probe tracking 
unit 608 and tracking cameras 604, 606 observe the markers 60 
602 on the probe, and thereafter computes the position and 
orientation of the tip of the probe. The tip position is 
transformed by the image data processor into the coordinate 
frame of the laser scanning unit or the real-time video 
camera 116. Due to the data registration technique of the 65 
present invention, the transformed positional data of the 
probe tip is then used to determine the position of the probe 



tip with respect to the coordinate frame of the anatomy 
image data (step 505). Accordingly, the registered image 
data is provided to a visualization output device, for 
example, the video mixer 124 and video monitor 126, so as 
to present to the surgeon a viewpoint of where the tip of the 
probe (which is hidden in the cavity) lies with respect to the 
full anatomical reconstruction of the patient (step 506). An 
exemplary visualization output of the present invention is 
provided in FIG. 7, which illustrates three cross-sectional 
slices through a full anatomical image data set of a skull, and 
a view of the full three-dimensional model of the skull with 
the position of the probe shown as P. 

The foregoing description has been set forth to illustrate 
the invention and is not intended to be limiting. Since 
modifications of the described embodiments incorporating 
the spirit and substance of the invention may occur to 
persons skilled in the art, the scope of the invention should 
be limited solely with reference to the appended claims and 
equivalents thereof. 

What is claimed is: 

1. An image data registration system comprising: 

an image data storage unit for storing a first data set of 
three-dimensional image data associated with a prede- 
termined portion of an object with reference to a first 
coordinate frame; 

a first image data acquisition and storage device for 
obtaining and storing a second data set of three- 
dimensional image data associated with a surface of 
said predetermined portion of said object with refer- 
ence to a second coordinate frame; and 

a second image data acquisition and storage device for 
obtaining and storing a third data set of three- 
dimensional image data associated with a movable 
probe positioned in close proximity to said predeter- 
mined portion of said object with reference to a third 
coordinate frame; 

an image data processor for registering said first,,second 
and third data sets of three-dimensional image data to 
generate a matched image data set of three-dimensional 
image data in which said first, second and third coor- 
dinate frames are relatively aligned with one another. 

2. The system of claim 1, wherein said object is an 
anatomical body. 

3. The system of claim 2, wherein said first data set of 
three-dimensional image data comprises three-dimensional 
anatomy image data. 

4. The system of claim 3, wherein said first image data 
acquisition and storage device comprises a surface scanning 
unit for scanning an outer skin surface of said predetermined 
portion of the body so as to obtain said second data set of 
three-dimensional image data. 

5. The system of claim 1, wherein said image data 
processor is operable for selecting a predetermined subset of 
data points from said third data set. 

6. The system of claim 5, wherein said image data 
processor is further operable for matching said subset of data 
points to all possible corresponding subsets of data paints 
associated with said second data set which results in a set of 
hypothesized transformation data sets. 

7. The system of claim 6, wherein said image data 
processor is further operable for sequentially mapping said 
second data set to each of said hypothesized transformation 
data sets to form a solution data set, said solution data set 
representative of a highly accurate transformation of said 
second data set into the third coordinate frame associated 
with said third data set. 

8. The system of claim 1, wherein said second data 
acquisition and storage device comprises a probe tracking 
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unit which tracks visual marks associated with said probe so 
as to obtain said third data set of three-dimensional data. 

9. The system of claim 8, wherein said image data 
processor determines position and orientation of said probe 
with respect to said first and second coordinate frames. 5 

10. The system of claim 9, wherein said image data 
processor determines and transforms the position and ori- 
entation of a tip of said probe with respect to said first and 
second coordinate frames. 

U. The system of claim 10 further comprising an image 
output device for producing an output image corresponding 
to the tip of said probe with respect to said matched image 
data set. 

12. A surgical image registration system comprising: 

a first storage device for storing three-dimensional 
anatomy image data associated with a predetermined 
portion of a patient's body with reference to a first 
coordinate frame; 

a surface scanning unit for scanning the skin surface of 
said predetermined portion of the patient's body so as 2Q 
to obtain three-dimensional surface image data; 

a second storage device for storing said three-dimensional 
surface image data with reference to a second coordi- 
nate frame; 

a probe tracking unit for tracking visual markers associ- 2 s 
ated with a movable probe positioned in close proxim- 
ity to said predetermined portion of the patient's body 
so as to obtain three-dimensional probe image data; 

a third storage device for storing said three-dimensional 
probe image data with reference to a third coordinate 30 
frame; and 

an image data processor operable for registering said 
three-dimensional anatomy image data, said three- 
dimensional surface image data and said three- 
dimensional probe image data to produce a matched 35 
image data set of three-dimensional image data in 
which said first, second and third coordinate frames are 
relatively aligned with one another. 

13. The system of claim 12 further comprising an image 
output device for producing an output image corresponding 40 
to a combination of said matched image data set. 

14. The system of claim 13, wherein said image output 
device comprises a video display monitor. 

15. The system of claim 13, wherein said image output 
device comprises a head-mounted display device. 45 

16. The system of claim 13, wherein said image output 
device comprises a surgical microscope. 

17. The system of claim 13, wherein said image output 
device comprises a transparent projection screen. 

18. The system of claim 12, wherein said three- 50 
dimensional anatomy image data comprises prerecorded 
magnetic resonance imaging (MRI) data. 

19. The system of claim 12 wherein said three- 
dimensional anatomy image data comprises prerecorded 
computed tomography (CT) data. 55 

20. The system of claim 12, wherein said three- 
dimensional anatomy image data is segmented into tissue 
types. 

21. The system of claim 12, wherein said surface scanning 
unit comprises a device for illuminating and reading a 60 
sequence of laser lines across the skin surface of said 
predetermined portion of the patient's body. 

22. The system of claim 12, wherein said image data 
processor is operable for selecting a predetermined subset of 
data points from said probe image data. 65 

23. The system of claim 22, wherein said image data 
processor is further operable for matching said subset of data 
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points to all possible corresponding subsets of data points 
associated with said surface image data which results in a set 
of hypothesized transformation data sets. 

24. The system of claim 23, wherein said image data 
processor is further operable for sequentially mapping said 
surface image data to each of said hypothesized transfor- 
mation data sets to form a solution data set, said solution 
data set representative of a highly accurate transformation of 
said surface image data into the third coordinate frame 
associated with said probe image data. 

25. The system of claim 12, wherein said image data 
processor determines position and orientation of said probe 
with respect to said first and second coordinate frames. 

26. The system of claim 25, wherein said image data 
processor determines and transforms the position and ori- 
entation of the tip of said probe with respect to said first and 
second coordinate frames. 

27. The system of claim 26 further comprising an image 
output device for producing an output image corresponding 
to the tip of said probe with respect to said matched image 
data set. 

28. A method of registering three-dimensional image data 
sets in a surgical imaging system, comprising: 

storing three-dimensional anatomy image data associated 
with a predetermined portion of a patient's body with 
reference to a first coordinate frame; 

scanning the skin surface of said predetermined portion of 
the patient's body so as to obtain three-dimensional 
surface image data; 

storing said three-dimensional surface image data with 
reference to a second coordinate frame; 

tracking visual markers associated with a movable probe 
positioned in close proximity to said predetermined 
portion of the patient's body so as to obtain three- 
dimensional probe image data; 

storing said three-dimensional probe image data with 
reference to a third coordinate frame; 

registering said three-dimensional anatomy data, said 
three-dimensional surface image data and said three- 
dimensional probe data to produce a matched image 
data set of three-dimensional image data in which said 
first, second and third coordinate frames are relatively 
aligned with one another; and 

producing an output image corresponding to a combina- 
tion of said matched image data set of three- 
dimensional image data. 

29. The method of claim 28, wherein said scanning further 
comprises illuminating and reading a sequence of laser lines 
across the skin surface of said predetermined portion of the 
patient's body. 

30. The method of claim 28, wherein said registering 
further comprises selecting a predetermined subset of data 
points from said surface image data. 

31. The method of claim 30, wherein said registering 
further comprises matching said subset of data points to all 
possible corresponding subsets of data points associated 
with said anatomy image data and producing a set of 
hypothesized transformation data sets. 

32. The method of claim 31, wherein said registering 
further comprises sequentially mapping said anatomy image 
data to each of said hypothesized transformation data sets to 
form a solution data set, said solution data set representative 
of a highly accurate transformation of said anatomy image 
data into the second coordinate frame associated with said 
surface image data. 

33. The method of claim 32, wherein said sequential 
mapping further comprises performing a least-squares mini- 
mization between said anatomy image data and said surface 
image data. 
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34. The method of claim 33, wherein said sequential 
mapping further comprises randomly perturbing said solu- 
tion data set and repeating said least -squares minimization in 
order to avoid local m inim a and solve for a global minima. 

35. The method of claim 28, wherein said registering 5 
further comprises selecting a predetermined subset of data 
points from said probe image data. 

36. The method of claim 35, wherein said registering 
further comprises matching said subset of data points to all 
possible corresponding subsets of data points associated 10 
with said surface image data and producing a set of hypoth- 
esized transformation data sets. 

37. The method of claim 36, wherein said registering 
further comprises sequentially mapping said surface image 
data to each of said hypothesized transformation data sets to 
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form a solution data set, said solution data set representative 
of a highly accurate transformation of said surface image 
data into the third coordinate frame associated with said 
probe image data. 

38. The method of claim 37, wherein said sequential 
mapping further comprises performing a least-squares mini- 
mization between said surface image data and said probe 
image data. 

39. The method of claim 38, wherein said sequential 
mapping further comprises randomly perturbing said solu- 
tion data set and repeating said least-squares minimization in 
order to avoid local minima and solve for a global minima. 

***** 
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